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DESCRIPTION 

IMAGE PROCESSING DEVICE AND METHOD THEREFOR AND PROGRAM 

CODES, STORING MEDIUM 

5 TECHNICAL FIELD 

The present Invention relates to an Image processing 
device and method for outputting the attitude or positional 
attitude of a measurement object, a program code, and a 
storage medium. 

10 

BACKGROUND ART INVENTION 

In recent years, a number of studies regarding a mixed 
reality ("MR") have been vigorously made. 

For the MR , there are provided a video see- through method 
15 in which an image in a real space photographed by a 

photographing apparatus such as a video camera is displayed 
with an image in a virtual space (e.g., a virtual object 
or the character information drawn by the computer graphics 
(hereinafter referred to as CG) ) superposed thereon, and 
20 an optical see- through method in which an image in the real 
space is optically transmitted to an HMD (Head-Mounted 
Display) mounted on the user's head and displayed with an 
image in the virtual space superposed on a display screen 
of the HMD. 

2 5 The applications of MR include the uses of medical 

assistance for presenting the interior of the patient ' s body 
to the physician as seeing through and the uses of work 



assistance by displaying the real things with an assembling 
procedure of products in the factory superposed thereon. 
Therefore, some new fields quite different in quality from 
the prior VR are expected. 
5 A common requirement for these applications is a 

technique for making alignment between the real space and 
the virtual space, for which various attempts have been made 
up to now. 

An alignment problem with the MR of video see- through 

10 method results in the problem of correctly attaining the 
positional attitude at the visual point of the photographing 
apparatus . Also , an alignment problem with the MR of optical 
see -through method similarly results in the problem of 
obtaining the positional attitude at the user ' s visual point . 

15 In the conventional MR system (particularly an indoor 

MR system) , a method of solving these problems typically 
involves deriving the positional attitude at the visual point , 
using a positional attitude sensor such as a magnetic sensor 
or a ultrasonic sensor. 

20 On one hand, the conventional MR system for outdoors 

makes use of a gyro sensor to derive the attitude at the 
visual point (more strictly, a three-axis attitude sensor 
consisting of a combination of a plurality of gyro sensors 
for measuring the angular velocities in the three axis 

25 directions and a plurality of acceleration sensors for 
measuring the accelerations in the three axis directions. 



as a matter of convenience, called a gyro sensor in this 
specification) . 

However, when a gyro sensor is used to obtain the 
attitude at the visual point, the gyro sensor of high 
precision has a drift error, so that a measurement error 
will occur in the azimuth direction along with the elapse 
of time. Also, the gyro sensor is only able to make the 
attitude measurements, and can not follow the changes in 
the visual point position. In other words , there may occur 
some dislocation between the real space and the virtual space 
along with the elapse of time or the changes in the visual 
point position. 



DISCLOSURE OF INVENTION 
The present invention has been achieved in the light 
of the above-mentioned problems, and it is an object of the 
invention to measure the attitude or positional attitude 
at the visual point, and more particularly to correct for 
an error in the azimuth direction component that may occur 
along with the elapse of time. 

To accomplish the above-mentioned object, an image 
processing device of the invention comprises, 

an image pick-up device having the fixed positional 
relation with a measurement object, 

an attitude sensor for measuring the attitude at an 
image pick-up visual point of said image pick-up device, 
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storage means for storing the calculation information 
to calculate the attitude and/or position of saidmeasurement 
object on the basis of an output from said attitude sensor, 
target image setting means for setting a target image 
5 that is an object for detecting a predetermined index on 
the basis of a picked-up image picked up by said image pick-up 
device , 

detecting means for detecting the position of said index 
in said target image by performing a template matching 
10 process between a template image of said index and said target 
image , 

updating means for updating said calculation 
information stored in said storage means on the basis of 
a detected position of said index detected by said detecting 
15 means, and 

calculation means for calculating the attitude and/or 
position of said measurement object on the basis of said 
measured value and said calculation information updated by 
said updating means . 

20 Also, said target image setting means obtains a 

prediction position of an index in said picked-up image, 
employing said measured value and said calculation 
information stored in said storage means . creates an image 
with a peripheral area around said prediction position in 

25 said picked-up image subjected to a rotational process on 
the basis of a rotational angle of said image pick-up device 
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in a roll direction that is derived from said measured value , 
and outputs said image as a target image. 

Other features and advantages of the present 
invention will be apparent from the following description 
5 taken in con junction with the accompanying drawings , in which 
like reference characters designate the same or similar parts 
throughout the figures thereof. 



BRIEF DESCRIPTION OF THE DRAWINGS 
10 The features and advantages of the present invention 

will more fully appear from the following detailed 
description of the preferred embodiments when read in 
connection with the accompanying drawings. 

FIG. 1 is a view showing an initial screen 1°; 
15 FIG. 2 is a diagram showing the configuration of the 

conventional image processing device; 

FIG. 3 is a diagram showing the configuration of an 
image processing device according to a first embodiment of 
the invention; 

20 FIG. 4 is a diagram showing a specific configuration 

of an operating module for corrected value of visual point 

position attitude 215 ; 

FIG. 5 is a flowchart showing a main process in the 

first embodiment of the invention; 
25 FIG. 6 is a flowchart for a specific process of creating 

a template image; 
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FIG. 7 is a flowchart for a specific process of 
calculating a correction matrixAM*; 

FIG. 8 is a flowchart for a correcting operation loop 
process in a second embodiment of the invention; 
5 FIG. 9 is a flowchart for a main process in a fourth 

embodiment of the invention; 

FIG. 10 is a flowchart for a specific process of 
obtaining a correction matrixAR 11 ; 

FIG. 11 is a flowchart for a main process in a fifth 
10 embodiment of the invention; 

FIG. 12 is a flowchart for a specific process of 
obtaining a correction matrixAT*; 

FIG. 13 is a flowchart for a main process in a seventh 
embodiment of the invention; 
15 FIG. 14A is a view for explaining a basic principle 

of template matching in this invention; 

FIG. 14B is a view for explaining a basic principle 
of template matching in this invention; 

FIG. 14C is a view for explaining a basic principle 
20 of template matching in this invention; 

FIG. 14D is a view for explaining a basic principle 
of template matching in this invention ; 

FIG. 14E is a view for explaining a basic principle 
of template matching in this invention; and 
25 FIG. 14F is a view for explaining a basic principle 

of template matching in this invention. 
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BEST MODE FOR CARRYING OUT THE INVENTION 
The preferred embodiments of the present invention will 
be described below in detail with reference to the 
accompanying drawings . 
5 [ Firs t embodiment ] 

In this embodiment, an image processing device for 
effecting presentation of an MR space without dislocation 
by correcting for an error of attitude measurement at the 
visual point of camera with an attitude sensor will be 

10 described below. 

FIG. 2 shows the configuration of the conventional image 
processing device for drawing an image of a real object with 
an image of a virtual object superposed thereon in accordance 
with an attitude of the HMD equipped with the attitude sensor. 

15 In FIG. 2, the HMD 200 relies on a video see -through 

method, and comprises a display 201 for displaying the image, 
a camera 202 (observer visual point camera) for picking up 
the image of the real space from the visual point position 
of the observer having this HMD 200 mounted, and an attitude 

20 sensor 203 (e.g. , a gyro sensor) for measuring the attitude 
at the visual point of the camera 202. Also, the image 
processing device 210 comprises a sensor input module 211 
for inputting a sensor output from the attitude sensor 203, 
an image input module 213 for inputting a photographed image 

25 of a real object from the camera 202, a visual point position 
attitude operating module 212 for generating the information 
( e . g . , 4 x 4 model view matrix M* ) representing the positional 
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attitude at the visual point of the camera 202 on the basis 
of the attitude at the visual point of the camera 202 input 
from the sensor input module 211 and the visual point position 
of the camera 202 obtained in other way, and an image 
5 generating module 214 for generating an image consisting 
of the image of the real object with the image of the virtual 
object superposed thereon on the basis of the information 
representing the positional attitude at the visual point 
operated by the visual point position attitude operating 

10 module 212 , and provides a presentation image on the display 
201. In this case, an image having dislocation that may 
be caused by an error accumulated in the sensor output along 
with the elapse of time is displayed on the display 201. 
In this embodiment, the visual point position is held 

15 in advance in the visual point position attitude operating 
module 212 as a fixed value. Generally, in the case where 
the distance to an observation object (real object, virtual 
object) in an MR space for observation is relatively large 
with respect to the actual movement amount of the visual 

20 point position, there is a nature that more or less error 
in the visual point position, if any, will not greatly affect 
the dislocation on the image. In particular, in the uses 
where the observation object exists far away such as an 
outdoor MR system, and the observer is standing at one place, 

25 it is effective to have the position of visual point as the 
fixed value . Of course , other positional sensor (e.g. , GPS ) 
for measuring the visual point position of the camera 202 
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is additionally attached to the HMD 200, its output may be 
input as the visual point position. 

In this embodiment , from the above reason , it is assumed 
that the error of the visual point position is fully small 
5 as the relative value , and the dislocation on the image caused 
by the error of the visual point position is sufficiently 
negligible . 

FIG. 3 shows the configuration of an image processing 
device in this embodiment to which the HMD 200 is connected. 

10 The like or same parts are designated by the same numerals 
as in FIG. 2, and are not described here. 

In FIG. 3, an image processing device 310 is one in 
which a corrected value operating module 215 is added to 
the image processing device 210 as shown in FIG. 2, and further 

15 comprises a visual point position attitude operating module 
312 that replaces the visual point position attitude 
operating module 212. This corrected value operating 
module 215 calculates the corrected value (correction matrix 
AM 1 ) , through a corrected value operation process as will 

20 be described later, on the basis of a photographed image 
input from the image input module 213 and the attitude at 
the visual point input from the sensor input module 211, 
and outputs it to the visual point position attitude 
operating module 312. The visual point position attitude 

25 operating module 312 performs an attitude correcting process 
as will be described later, on the basis of the attitude 
at the visual point of the camera 202 input from the sensor 
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input module 211, the visual point position of the camera 
202 obtained by other method, and the corrected value input 
from the corrected value operating module 215, correcting 
the positional attitude information (a model view matrix 
5 M*) calculated on the basis of the sensor output, and 
generating a visual point position attitude information 
after correction (a corrected model view matrix M?*). 

A basic principle of the corrected value operation 
process in the corrected value operating module 215 will 

10 be described below. 

The corrected value operation process is basically 
performed on the basis of the observation prediction position 
of the landmark on the image predicted from the sensor output 
and the observation position on the image of the landmark 

15 actually detected by the image processing, employing a 
landmark (e.g., a real object (or a part) that can use the 
features of an image for its projected image as an index 
of alignment such as the corner of the building or the roof 
of the house) . Accordingly, it is the maximal point of the 

20 corrected value operation process how to detect the 

observation position of landmark from the image correctly 
and stably. 

In this embodiment , the landmark is detected by template 
matching employing a template image of landmark. 
25 Generally, in the case of extracting the image features 

from the image by template matching, the rotation of image 
features on the image screen is problematical. This 



rotation of image features is caused when the camera or the 
photographing object is rotated in a roll direction in the 
camera coordinate system. For example, in the case that 
a landmark L is detected from a photographing image I as 
shown in FIG. 14B, employing a template image T as shown 
in FIG. 14A, the landmark can not be detected stably, if 
a search process without regard to the rotation of image 
features is performed . On one hand , a plurality of template 
images T ' are prepared which is obtained by rotating the 
template image T (by every 45 degrees in a shown example) 
in view of rotation of image features, as shown in FIG. 14C, 
and the search process is performed for each template image 
to detect the landmark corresponding to the rotation of image 
features. However, the amount of calculation is increased 
in proportion to the number of templates, resulting in very 
high computational load. 

In this embodiment, the attitude of the visual point 
of the camera 202 is measured by the attitude sensor 203. 
Of the measured values, a value in the azimuth direction 
has an error accumulated along with the elapse of time, as 
previously described, but for two axes (i.e. , roll direction 
and pitch direction) other than the azimuth direction, the 
relatively correct values are acquired. Accordingly, a 
transformed image I ' having the photographed image I rotated 
is generated on the basis of the rotational angle in the 
roll direction of the camera 202 that is measured by the 
attitude sensor 203, as shown in FIG. 14D, and the search 
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process with the template image T on this image I ' is performed 
to detect a landmark not dependent on the rotation of image 
features . 

Further, in this embodiment, the measured values by 
the attitude sensor 203 have been obtained for the attitude 
of other two axes , and the attitude corrected value in the 
former frame has been obtained in the attitude corrected 
value operating module 215 through the process up to the 
former frame. Accordingly, a rough position p of landmark 
on the photographed image I can be predicted based on those 
values, as shown in FIG. 14E, and the rotation process is 
performed only in the neighborhood area (a target image 
extracting area in the figure ) around the prediction position 
to produce a target image R that is subjected to the search 
process of landmark (FIG. 14F), limiting the search range. 

Accordingly, the landmark can be detected by template 
matching fast and stably. 

A specific configuration of the corrected value 
operating module 215 is shown in FIG. 4. 

The corrected value operating module 215 comprises a 
template image generating module 403 for generating a 
template image on the basis of an image 1° as will be described 
later, a target image generating module 404 for generating 
a target image on the basis of an image I* at time t and 
the attitude (roll*) of the camera 202, a correspondence 
searching module 402 for calculating the similarity using 
the target image and the template image and detecting the 
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position of landmark, and a corrected value updating module 
401 for updating the corrected value to the latest corrected 
value (a correction matrix AM*) in accordance with the 
position of detected landmark and outputting it. 
5 The variables for use in this embodiment will be 

described below. 

The i-th landmark (i= 1, 2, 3, . .) is L t . 

The position (known) in the world coordinates of 
landmark L ± is P ± = (Xi, Y i# Z A , 1) T . 
10 The predetermined position of camera is (X 0 , Y° r Z°). 

The predetermined attitude of camera for use in 
generating the template image is (roll 0 , pitch 0 , yaw 0 ). 

The model view matrix (transformation matrix from the 
world coordinate system to the camera coordinate system) 
15 in the predetermined positional attitude of camera is M°. 

The focal length (known) of camera is f . 

The projection transformation matrix (transformation 
matrix from the camera coordinate system to the image 
coordinate system) of camera is S. 
20 The photographed image in the predetermined positional 

attitude of camera is 1°. 

The photographing position on the image 1° of landmark 
^ is piO = ( Xl V, yiV 0 , h ± °)\ 

The template image for retrieving the landmark L ± is 

25 T ± . 

The size (predetermined) of template image is N x N. 
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The range of coordinates of the template image is xs T , 
xe T , ys T , ye T (with the fraction part of xs T = ys T = -N/2 
rounded down. xe T = ye T = xs T +N-1). 

The image photographed at time t is I t . 
5 The attitude value measured by the sensor at time t 

is (roll 1 , pitch 1 , yaw 1 ). 

The model view matrix (transformation matrix from the 
world coordinate system to the camera coordinate system) 
calculated from the attitude measured value (roll 1 , pitch 1 , 
10 yaw 1 ) is M 1 . 

The photographing prediction position of landmark Li 
on the image I 1 is pt 1 = (x^, y^hi 1 , hi 1 ) 7 . 

The image pick-up position of landmark Li actually 
detected on the image I 1 is p?! 1 = (xSi 1 , ySi 1 ). 
15 The target image subjected to the search process for 

detecting the landmark Li from the image I 1 is Ri 1 . 

The search range (predetermined) for landmark in the 
x direction is ±m. 

The search range (predetermined) for landmark in the 
20 y direction is ±n. 

The size of target image is N'x N" (N ' = N+2m, N" = 
N+2n). 

The range of coordinates of the target image is xs R/ 
xe Rl ys R , ye R (xs R = xs T - m, xe R = xe T + m, ys R = ys T - n, 
25 ye R = ye T + n) . 

The detected coordinates of landmark L± on the target 
image Rt 1 is (ji 1 , kj. 1 ) . 
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The typical values of detected coordinates ( ji*, ki*) 
of each landmark are (j*, k*). 

The corrected updated value of the camera attitude 
calculated at time t is Aroll, Apitch, Ayaw. 
5 The corrected updated value of the camera position 

calculated at time t is Ax, Ay, Az. 

The correction matrix for correcting the model view 
matrix M t calculated at time t is AM*. 

The correction matrix (at time t-1) already calculated 
10 by the prior process is AM*" 1 . 

The corrected updated matrix for updating the 
correction matrix AM*" 1 to the correction matrix AM* is AM'*. 

The model view matrix after correction which is M* 
corrected by the correction matrix AM* is M$*. 
15 The model view matrix after correction which is M* 

corrected by the correction matrix AM*" 1 is M'*. 

The correction process for an error of attitude 
measurement in this embodiment will be described below in 
accordance with the processing flow on the basis of the above 
20 settings. 

<Creating the template image > 

Firstly, the camera for photographing the real space 
is set at a predetermined positional attitude to photograph 
the image I 0 . An example of the image I 0 is shown in FIG. 
25 1. In the same figure, Li to L 4 are landmarks, the frame 
sections as indicated by Tx to T 4 being extracted as the 
template images corresponding to the respective landmarks . 
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Next, the model view matrix M° is calculated. A 
calculation method for calculating the model view matrix 
from the position and attitude of camera is well-known, and 
is not described herein. 
5 Also, pi 0 is calculated for each landmark (Li to L 4 in 

FIG. 1) in the photographed image in accordance with the 
following Equation. 

And a template image Ti (in the section as indicated 
10 by T x to T 4 in FIG. 1) for each landmark is created by a 
method as will be described later. 

When roll 0 is 0, a rectangular area having a size of 
N x N centered at (xi°, yi°) is extracted from the image I 0 , 
and this may be the template image T ± . In the template image 
15 T ir if the center of image is represented as the coordinates 
(0, 0) , this process can be described in the following way. 
TiU, k) = I^X! 0 + j, yi ° + k) 
Where j = xs T to xe T , k = ys T to ye T ° 
On the other hand, when roll 0 is not 0, a rectangular 
20 area which is the rectangular area of N x N rotated by -roll 0 
at a center of (xi°, yi 0 ) is extracted. That is, the template 
image T ± is created such that for each pixel, j = xs T to 
xe T , k = ys T to ye T 

T ± (j, k) m I°(x 1 ° + jcos(-roll°) - ksin( -roll 0 ) , yi ° + 
25 jsin(-roll°) + kcos( -roll 0 ) ) 

Calculating the model view matrix M* at each time> 
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The model view matrix M 1 is calculated on the basis 
of the sensor output (attitude (roll 1 , pitch 1 , yaw 1 ) ) at time 
t and the predetermined camera position (X°, Y° , Z°). A 
calculation method for calculating the model view matrix 
5 from the position and attitude of camera is well-known, and 
is not described herein . 

<Corrected value operation process : calculation of the 
correction matrix AM 1 for correcting the model view matrix 
M t > 

10 A calculation method for calculating the correction 

matrix AM 1 for correcting the model view matrix M 1 will be 
described below. 

Firstly, the model view matrix M 1 is corrected, 
employing the correction matrix AM 1 " 1 already calculated in 
15 the prior process, calculating the matrix M fl . When this 
process is performed for the first time (in case of t = 0) , 
the correction matrix AM 1 " 1 is a unit matrix. 
M rl = AM 1 " 1 ** 1 

Pi 1 is calculated for each landmark in accordance with 
20 the following Equation. 
Pi 1 = SM^Pi 

This method is well known, and not described in detail . 
As a result of calculating the coordinates of each landmark, 
the landmark having the coordinates outside the coordinate 
25 range of the image I 1 is excluded from the following 
processing. 
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Next . the target image is created for each landmark . 
Specifically, a rectangle which is the rectangular area of 
«• x ,- rotated by - ro n< at a center of ^ y±t) ^ ^ 

the local coordinates of the same image from the image I* 
> is extracted. 

That is, the following transformation is made for each 
pixel, j = XSt to xe T , k = ys T to ye T 

*S (3. k) = i* (Xl t + joo.(-roll*) - ksin(-roll*), y± t 
+ Jsin(-roll t ) + kcosC-roll*)) 

Then, for each landmark, the target image R t * and the 
template image T ± are matched to obtain the landmark position 
Ui*. *t<> on the target image. A specific processing of 
a method for obtaining the landmark position will be 
described below. 

First of all, the similarity e(j. k) between the 
rectangular area centered at the coordinates (J, k, on the 
target image and the template image Ti is calculated. 
The calculation of the similarity can be made by 
cross-correlation or SSD (Sum of Squared Difference) , for 
example, but may be made by any of the well-known template 
matching methods. This similarity e(j, k) is calculated 
for all j and k (j = -m to m, k = -„ to n) , and (J , fc) fQr 
the maximum similarity e(j, k) is k ± '). 

And the typical value (J*. k *, is calculated from (J±t> 
k^) obtained for each landmark. The calculation of the 
typical value can be made by calculating the average value 
or median value of klM obta±ned fQr each 
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Using only the landmarks having the similarity e( , k**) 
larger than a predetermined threshold for ( j t \ ki *) obtained 
for each landmark, the detected results with low reliability 
can be excluded. In this case, if the number of landmarks 
having the similarity eCL*. k± *) larger than or equal to 
the threshold is a predetermined number or less, the 
corrected value operation process at time t may be aborted. 

Then, the correction matrix AM* is updated on the basis 
of the detected result of landmark. 

First of all, the corrected updated values Aroll. Apitch, 
Ayaw of the camera attitude is obtained in the following 
way. 

Aroll = o 

Apitch = arctan (k*/f ) 
15 Ayaw = arctan (j*/f) 

Since it is assumed that the camera position is fixed, 
the corrected updated values of position Ax, Ay, Az are all 
0. 

Next, the corrected updated matrix AM'* is calculated 
as the model view matrix that is determined by the attitude 
Aroll. Apitch, Ayaw, and the position Ax, Ay, Az. A 
calculation method for calculating the model view matrix 
from the position and attitude of camera is well-known, and 
is not described herein. 

And the correction matrix AM* after update is calculated 
from the correction matrix AM* 1 and the corrected updated 



20 
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matrix AM ' 1 as obtained above in accordance with the following 
Equation . 

AM* = AM #t AM 1 " 1 

<Positional attitude correcting process: calculating 
5 the model view matrix M$ t after correction) 

The model view matrix M$ t after correction at time t 
can be obtained in accordance with the following Equation. 
M$* = AM^ 

And if the model view matrix M$ t after correction is 
10 used to draw and display CG, the dislocation in the azimuth 
direction along with the elapse of time can be relieved even 
by the use of a gyro sensor. 

The flowcharts of FIGS . 5 to 7 for the correction process 
in this embodiment will be described below. 
15 FIG . 5 is a flowchart of a main routine of the correction 

process . 

Firstly, a template image is created (step S501). A 
flowchart for the specific process of creating the template 
image is shown in FIG. 6. 

20 First of all, an image 1° is input from the camera fixed 

at the predetermined positional attitude ( step S601 ) . Then 
the model view matrix M° is calculated on the basis of the 
positional attitude of camera at this time ( step S602 ) . Then 
pi° is calculated for all i (or all landmarks) (steps S603, 

25 S604) . Then, the template image is created. The creation 
method involves, as above described, calculating the pixel 
values for all j , k within the above range for each landmark. 
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and storing them in the coordinates ( j , k) of the template 
image T ± (steps S606 to S608). 

If the template image is created in accordance with 
the procedure as shown in FIG. 6, the control returns to 
5 the routine of FIG. 5, and an image I* is photographed (step 
S502). Then, the sensor output is acquired (step S503). 
The steps S502 and S503 are not performed in this order , 
but may be reversed, or performed concurrently in 
synchronism . 

10 Then the model view matrix M t is calculated on the basis 

of the sensor output ( step S504 ) . And the correction matrix 
AM* is calculated (step S505) . A flowchart for the specific 
process in calculating the correction matrix AM* is shown 
in FIG. 7, and described below. 

15 Firstly, the model view matrix M* is corrected by the 

correction matrix AM 11 " 1 to obtain the model view matrix M ft 
(stepS701) . And Pi 11 is calculated for all I or all landmarks 
(steps S702, S703). The calculated p^, which is outside 
the range of the image I t , is excluded from the following 

20 process. 

Then, the target image Ri* is calculated for each 
landmark (steps S704 to S706). And the matching between 
the target image R 1 t and the template image T t is made to 
calculate the similarity e(j, k) for each of j , k (steps 
25 S707, S708). And (j, k) at which the similarity e(j, k) 
is maximum is made (ji*, k^) (step S709) . The above process 
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from step S707 to S709 is performed for all i or all landmarks 
(step S710) . 

The average value of (ji*, kj*) obtained is computed 
to obtain (j*, k t ) (step S711). Also, the corrected value 
5 of the positional attitude of camera is obtained ( step S712 ) , 
and the corrected updatedmatrix AM ' t is obtained ( step S 71 3 ) , 
and lastly the correction matrix AM* is obtained ( step S714 ) . 

The correction matrix AM 11 is calculated in accordance 
with the procedure as shown in FIG. 7. Then the control 
10 returns to the routine of FIG. 5 to correct the model view 
matrix M* using the calculated correction matrix AM 1 (step 
S506) . 

And CG is drawn and displayed using the model view matrix 
after correction MS* (step S507). 
15 As described above, the image processing device and 

method of this invention allows the MR without dislocation 
to be implemented by correcting for an error of attitude 
measurement at the visual point of camera with the attitude 
sensor. 

20 [Second embodiment] 

In the first embodiment, the correction process is 
performed in a single loop (drawing loop). In this case, 
it is not possible to acquire sufficiently the frame rate 
of drawing due to the computational load of the image 

25 processing. In other words, if the image processing is 
simplified (or reduced in computation amount) to secure the 
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frame rate of drawing, the correction precision can not be 
sufficiently attained* 

Thus, in this embodiment, the drawing loop and the 
correction operation loop are divided and operated at 
5 independent update periods (e.g., the drawing loop is 60 
Hz and the correction operation loop is 1 loop/ second) . Also , 
the processing device in this embodiment may be the same 
image processing device as used in the first embodiment. 

<Drawing loop> 

10 Basically, the processing is performed in accordance 

with the flowcharts as shown in FIGS. 5 and 6, except that 
the latest correction matrix AM S passed from a correction 
operation loop as will be described later is obtained to 
have AM* at step S505 . 

15 <Correction operation loop> 

FIG. 8 shows a flowchart of a correction operation loop 
process. Firstly, the image I s at time s and the model view 
matrix M s at that time are input from the drawing loop (step 
S801) . The correction matrix AM S is calculated in the same 

20 manner as at step S505 as described in the first embodiment 
(step S802). And the calculated correction matrix AM S is 
sent to the drawing loop (step S803). The above process 
is repeated until an end permission is given (step S804) . 
In this embodiment , the drawing loop and the correction 

25 operation loop are divided, and performed in one image 
processing device (as an example), but these loops may be 
performed in separate computers. And these loops are in 
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conununicat ion between those computers to transmit or receive 
the processing results . In this way , the number of processes 
for each computer to handle is reduced, resulting in faster 
processes . 

[Third embodiment] 

In the second embodiment, in the correction process 
of the model view matrix, the model view matrix after 
correction M$ t is obtained by a simple product operation 
between the correction matrix AM* and the model view matrix 
M* through the use of the sensor. However, since updating 
the correction matrix occurs at a larger interval than the 
drawing period, it can not be said that the correction matrix 
necessarily represents the appropriate correction 
information at the current frame (time t) . 

Thus, in this embodiment, the correction matrix AM 1 
appropriate at time t is calculated, employing the past 
correction matrix obtained through the correction operation 
loop at step S505 in the second embodiment . 

Firstly, the correction matrix AM S obtained at time 
s is expanded to calculate the corrected value Ayaw s in the 
azimuth direction of the camera attitude and the corrected 
value Apitch* in the pitch direction. The method for 
obtaining individual rotational components from the model 
view matrix is well-known and not described herein. The 
same processing is performed at time s-1, and the corrected 
values Ayaw* and Apitch* of the camera attitude at time t 
are obtained in the following way. 
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Ayaw* = Ayaw s + (Ayaw s - Ayaw 5 ' 1 ) Ast/As 
Apitch* = Apitch* + (Apitch* - Apitch^) x Ast/As 
Herein, Ast is the elapsed time from time s to time 
t. and As is the elapsed time from time s-1 to time s. 

And employing the corrected values Ayaw* and Apitch* 
thus obtained, the correction matrix AM* is calculated. As 
a result . the correction matrix appropriate for the current 
frame (time t) can be calculated by employing a calculation 
method of the correction matrix in this embodiment. 

In this embodiment, the extrapolation of corrected 
value is effected by the linear prediction of first order 
as shown in the above Equation, but the prediction method 
of corrected value is not limited thereto, and the linear 
prediction of second order or other prediction methods may 
15 be employed. 

[Fourth embodiment J 

In this embodiment , a method for making correction more 
accurately than in the first embodiment will be described 



10 



below. 

20 



First of all, the variables used in this embodiment 
different from those of the above embodiments will be 
described below. 

Rotational component of the model view matrix based 
on the sensor output at time t is R*. 

Parallel movement component of the model view matrix 
based on the predetermined position of camera is T\ 
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Detected position of landmark L ± on the image I* is p?^ 
= (x$ i t / y$ i t ). 

Position of "projected point onto the image I tn for 
the landmark L t in the camera coordinate system is pc**. 
5 Corrected updated matrix of the model view matrix 

obtained from the landmark Li (rotational component in the 
azimuth direction) is ARt'* 

Corrected updated value in the yaw direction obtained 
from the landmark L t is AyaWi*. 
10 Corrected updated value in the yaw direction obtained 

from all landmarks is Ayaw*. 

Correction matrix of the model view matrix (rotational 
component in the azimuth direction) is AR*. 

Correction matrix already calculated in the prior 
15 process is AR 11 " 1 (a unit matrix in the first loop). 

Rotational component of the model view matrix corrected 
by the correction matrix AR t_1 is R ft . 

Model view matrix corrected by the correction matrix 
AR*' 1 is M ft . 

20 Corrected updated matrix for updating the correction 

matrix AR t_I to the correction matrix AR t (rotational 
component in the azimuth direction) is AR ' t . 

A correction method on the basis of the above settings 
in this embodiment will be described below with reference 

25 to FIGS. 9 and 10 showing the flowcharts of the processing 
for the same method. 
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FIG. 9 is a flowchart of a main process in this embodiment . 
The processing from step S901 to step S903 is the same as 
that from step S501 to step S503 in the first embodiment, 
and not described here. 
5 Then , the rotational component R* and parallel movement 

component T< of the model view matrix are calculated (step 
S904). Specifically, the rotational component R* is 
calculated on the basis of the sensor output (or the attitude 
of camera obtained from the sensor output) (roll*, pitch 1 . 

0 yaw^byawell-knownmethod. On the other hand , theparallel 
movement component T* is calculated on the basis of the visual 
point position of camera by a well-known method. 

And the correction matrix AR* is obtained (step S905) . 
FIG. 10 shows a flowchart of the specific processing for 

1 calculating the correction matrix AR*. which will be 
described below. 

Firstly, the matrix R* is corrected with the correction 
matrix AR 1 " 1 already calculated in the prior process to obtain 
the matrix R ft . 

R'* = R* AR*" 1 

Next, the matrix M'* is obtained in the following way, 
using the matrix R rt (step S1001). 
M 't = R .t T t 

The processing from step S1002 to step S1010 is the 
same as that from step S702 to step S710 in the above embodiment , 
and not described here. 
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Then, the position p?^ = (x$ i t , y$ i t ) of each landmark 
on the image I 1 is calculated using (jt 1 , ki*) obtained (step 
S1012). This calculation is made in accordance with the 
following Equation. 

x$ 1 t = Xi* + j^cos (-roll 1 ) - k^sin (-roll 1 ) 
ySi* = Yi* + j^sin (-roll 1 ) + k^cos (-roll 1 ) 
And the position pcj 1 of "projected point onto the image 
I lw for the landmark in the camera coordinate system is 
calculated (step S1013). 

pci 1 = (x$ i 1 , y$i t , -f, 1) T 

Then, supposing that a is a scaling parameter, pc^-a 
= R ' t AR 1 ' 1 T 1 P i holds. Solving this equation, Ayaw* 1 is 
calculated. This method is shown below. In the following, 
Inv(M) is an inverse matrix of the matrix M. 

Assuming that 

PSi* = (X$i\ Y$ i t , Z$ ± \ 1) T = Inv (R ' 1 ) pc^ 
P'i = (X'i, Y' ±/ Z' i# 1) = T'Pi 
Consequently , 
PSi* = ARi'^'i/a 
Therefore, 

X$ 1 1 = {cos (Ayawi 1 ) X' t - sin (Ayawi 1 ) Z' ± }/a 
Z$ i t = {sin (Ayawi 1 ) X'i + cos (Ayawi 1 ) Z\}/sl 
Solving the above equations, 

Ayawt 1 = arctan {(Z$ 1 t -X' 1 - X$ i t Z ' 1) / (X$ 1 1 -X' t + Z$ i t -Z\)} 
(step S1014 ) . The processing at step S1014 is repeated 
for all I , or for all landmarks ( step S1015 ) . And the average 
value Ayaw 1 of AyaWi 1 for all I is obtained (step S1016). 



- 29 - 



And the corrected updated matrix AR ' t is obtained using 
the corrected updated value Ayaw* (step S1017). A method 
for calculating the model view matrix to rotate the 
coordinate system in the azimuth direction by an arbitrary 
5 angle (here Ayaw*) is well known, and not described here. 
Employing this corrected updated matrix AR ' t , the correction 
matrix AR* is obtained in the following way (step S1018) . 

AR 1 = AR t_1 AR ft 

After the correction matrix AR* is calculated in 

10 accordance with the procedure as shown in FIG . 10 , the control 
returns to the main routine of FIG. 9. Then the model view 
matrix M$ t is calculated using the correction matrix AR* 
calculated (step S906). This calculation is made in 
accordance with the following Equation. 

15 M$ t = R* AR t T t 

And CG is drawn and displayed using the model view matrix 
calculated in the same manner as in the first embodiment 
(step S907) . 

[Fifth embodiment] 

20 In the first to fourth embodiments, it is assumed that 

the visual point position is known, and the attitude 
(direction, angle) is only corrected. As described 
previously, in the case that the distance to the observation 
object is relatively greater than the movement amount of 

25 the visual point position, it is effective that the visual 
point position is a fixed value . However, if that assumption 
is invalid, the dislocation is caused by the movement of 
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the visual point. Hence, in this embodiment, a method for 
correcting the visual point position is described. In this 
embodiment, it is assumed that the movement amount ATz in 
the Z axis direction (i.e. , the depth direction , or direction 
5 perpendicular to the photographing place) in the camera 
coordinate system is always 0. Also, for the rotational 
component, it is assumed that the correct value has been 
acquired by the sensor. If this assumption holds, the 
position can be corrected by detecting one landmark at 
10 minimum. 

Herein, the settings in this embodiment are listed in 
the following. 

Rotational component of the model view matrix based 
on the sensor output at time t is R*. 
15 Parallel movement component of the model view matrix 

based on the predetermined position of camera at time t is 
T*. 

Correction matrix of the model view matrix (parallel 
movement component in the world coordinate system) is AT*. 
20 Correction matrix of the model view matrix obtained 

from the landmark Li (parallel movement component in the 
world coordinate system) is ATi* . 

Correction matrix already calculated in the prior 
process is AT*' 1 (a unit matrix in the start loop). 
25 Parallel movement component of the model view matrix 

corrected by the correction matrix AT t_1 is T ft . 
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Model view matrix corrected by the correction matrix 
AT*" 1 is M'*. 

Corrected updated matrix of the model view matrix 
(parallel movement component in the camera coordinate 
5 system) is ATc* . 

Corrected updated value in the x axis direction (in 
the camera coordinate system) obtained from the landmark 
L ± is ATxi*. 

Corrected updated value in the y axis direction (in 
10 the camera coordinate system) obtained from the landmark 
L ± is ATyj*. 

Corrected updated value in the x axis direction (in 
the camera coordinate system) obtained from all landmarks 
is ATx*. 

15 Corrected updated value in the y axis direction (in 

the camera coordinate system) obtained from all landmarks 
is ATy*. 

A correction method on the basis of the above setting 
in this embodiment will be described below with reference 
20 to the FIGS . 11 and 12 showing the flowcharts of the processing 
for the same method. 

FIG. 11 is a flowchart of a main process in this 
embodiment. The processing from step S1101 to step S1104 
is the same as that from step S901 to step S904 in the fourth 
25 embodiment, and not described here. 
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Then, the correction matrix AT t is obtained (step S1105) . 
FIG. 12 shows a flowchart of the specific processing to obtain 
the correction matrix AT*, which will be described below. 
Firstly, the matrix T t is corrected with the correction 
5 matrix AT t_1 already calculated in the prior process, and 
the matrix T ' t and the matrix M ' t are obtained in the following 
way (step S1201). 

M rt = R^T'* 

10 The processing from step S1202 to step S1211 is the 

same as that from step S1002 to step S1012 in the fourth 
embodiment , and not described here . 

Then, at stepS1212 / the corrected updated values ATxi* , 
ATyj* with regard to the landmark Li is calculated. 
15 ATXi 1 = f Zc^ (x$ i t - Xi*) 

ATy^ = f-ZCi* (y$± t - Yi*) 

Herein, Zc^ is the z coordinate of landmark in the camera 
coordinate system, and has the value in the third component 
of M ft Pi. 

20 The above corrected updated values ATxi*, ATy^ are 

obtained for all i, or all landmarks (step SI 2 13) , and the 
average value ATx* , ATy* of the corrected updated value ATXi* , 
ATyi 1 for all i is obtained (step S1214) . And employing the 
average value ATx fc , ATy* of the corrected updated values 

2 5 obtained, the corrected updated matrix ATc t for making 

parallel movement of ATx* in the x direction and ATy* in 
the y direction in the coordinate system is calculated (step 
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S1215). A method for calculating the coordinate 
transformation matrix for making arbitrary parallel 
movement to the coordinate system is well known, and not 
described here. And the correction matrix AT* is obtained 
5 in the following way (step S1216). 
AT* = Inv (R*) ATc t R t AT t " 1 

After the correction matrix AT 1 is calculated in 
accordance with the process as shown in FIG. 12, the control 
returns to the main routine of FIG . 11 . The model view matrix 
10 M$* is calculated employing the calculated correction matrix 
AT* (step S1106). This calculation is made in accordance 
with the following Equation. 

M$ t = R t AT t T t 

And CG is drawn and displayed employing the model view 
15 matrix calculated in the same way as in the first embodiment 
(step S1107) . 

[Sixth embodiment] 

In the fifth embodiment, ATz is assumed to be always 
0, and in the case where the visual point position is moved 

20 forth or back in the visual line direction, the correct 
alignment can not be effected. In this embodiment, it is 
possible to deal with the case that ATz is not 0 by observing 
two or more landmarks at all times . 

The flowchart of the correction process in this 

25 embodiment is basically the same as those of FIGS. 11 and 
12 in the fifth embodiment, except that the contents of 
processing at steps S1214 and S1215 are different. The 
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processing of steps S1214 and S1215 for the correction 
process in this embodiment will be described below. 

Supposing that the corrected updated values in the x # 
y, and z direction in the camera coordinate system are ATX*, 
5 ATyV ATz*, the following Equation holds between the image 
pick-up prediction position p 1 t and the detected position 
p$ 1 t of landmark for each landmark. 

ATx* + x$ 1 t f-ATz t = f-Zc/ (x$ i t - Xj*) 

ATy 1 + y$ 1 t f-ATz t = f-Zc^ (y$± t - yi*) 
10 Therefore, the following simultaneous equations are 

set up for a plurality of landmarks and solved to calculate 
the unknown corrected updated values ATx 11 , ATy* , ATz t (step 
S1214) . 
[Equation 1] 
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And employing the calculated values ATx* , ATy*, ATz* , 
the corrected updated matrix ATc* is obtained by a well-known 
20 method (step S1215). And at step S1216, the correction 
matrix AT* is obtained employing the updated matrix ATc* 
obtained in the same manner as in the fifth embodiment. 
[ Seventh embodiment ] 
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In the first to sixth embodiments, any one of the 
rotation and the parallel movement can be only corrected. 
In this embodiment, both the rotation and the parallel 
movement are corrected. A basic method involves correcting 
5 the parallel movement after correcting the rotation. 

However, the method is not limited thereto, and may involve 
correcting the parallel movement and the rotation in reverse 
order, or repeating correction of parallel movement by a 
fixed number after correcting the rotation (or in reverse 
10 order ) , or until the error is reduced below a preset threshold 
or the error variation due to correction is reduced below 
a threshold. 

Herein, the settings for use in this embodiment are 
listed below. 

15 Rotational component of the model view matrix corrected 

by the correction matrix obtained at the intermediate stage 
of process is R Mt . 

Model view matrix corrected by the correction matrix 
obtained at the intermediate stage of process is M Ht . 

20 A correction process in this embodiment on the basis 

of the above settings will be described below. 

FIG. 13 shows a flowchart of a main process in this 
embodiment. The flowchart in the same figure has a process 
(step S1306 ) for calculating the correction matrix AT* added 

25 to the flowchart as shown in FIG . 9 f or the fourth embodiment , 
in which the process (step S1305) for calculating the 
correction matrix AR* is different. In the following, the 
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process (step S1306) for calculating the correction matrix 
AT* and the process (step S1305) for calculating the 
correction matrix AR* in this embodiment will be described. 
Other parts are not described here. 
5 The flowchart of the specific processing to calculate 

the correction matrix AR 1 at step S1305 is fundamentally 
the same as that of FIG. 10 in the fourth embodiment, except 
that the matrix R #t and M ft B as well as the matrix T rt , are 
calculated at step S1001 in this embodiment. 
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And the derived T #t is employed instead of the fixed 
value T* in FIG. 10 in the following steps (e.g., S1014). 
15 On the other hand, the flowchart of the specific 

processing to correct the correction matrix AT* at step S1306 
is fundamentally the same as that of FIG. 12 in the fifth 
embodiment, except that at step S1201, the matrix R* is 
corrected employing the correction matrix AR* obtained at 
20 step S1305, and the matrix R' ft and M Mt are obtained in 
accordance with the following Equations. 

R Mt = R t AR t 

M nt = R nt T ' t 

In the process of this embodiment, ATxi* and ATy^ are 
25 obtained at step S1212 in the flowchart of FIG. 12 in 
accordance with the following Equations. 
ATXi* = f-Zc i t (x$ i t - xj.*) 
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ATy^ - fZc t t (y$ ± t - yj) 

Herein , Zci* is the z coordinate of landmark in the camera 
coordinate system, and has its value in the third component 
of M nt Pi. 

5 Also, in the process of this embodiment, the correction 

matrix AT* is calculated at step S1216 in the flowchart of 
FIG. 12 in accordance with the following Equation. 
AT* = Inv(R wt ) ATc t R nt AT t_I 

And if the calculation of the correction matrix AT* 
10 is ended, the control returns to the flowchart as shown in 
FIG. 13. At step S1307, the model view matrix M?* is 
calculated in the following way. 
M$ t = R t AR t AT t T t 

The process (steps S1305, S1306) for obtaining the 
15 correction matrix AR* , AT* may be repeated by a predetermined 
number of times, as described above. 
[Eighth embodiment] 

In the first to seventh embodiments, the position of 
landmark in the world coordinate system is known, but may 

20 be determined by other methods. That is, the position of 
landmark may be directly specified on the image 1° at an 
initial positional attitude, or the feature points with 
remarkable (easily traceable) image features (e.g., edge 
part or texture part ) may be extracted from the image 1° 

25 at the initial positional attitude , this position being made 
the position of landmark. 
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Herein, an instance is considered in which the image 
feature picked up at the image coordinates (Xi°, y±°) are 
designated or detected by manual input or image processing 
and used as the landmark Li. Assuming that the camera 
5 coordinate of this landmark is PCi° = (Xi°, yi 0 , f, 1), the 
world coordinate can be defined as Pi = Inv(M° ) Pcj 0 , employing 
an inverse matrix of the model view matrix M° at the initial 
position attitude, and the methods as described in the first 
to third embodiments can be directly employed. 

10 Since the information of landmark position in the depth 

direction can not be obtained, the correction can not be 
made employing the depth information of landmark position 
(position correction as described in the fifth embodiment 
and beyond) . 

15 [Modification 1] 

In the above embodiments, the attitude (or positional 
attitude) at the visual point of camera in the MR system 
is measured . However , this invention is not limited thereto , 
and may be applied to any uses for measuring the attitude 

20 (or positional attitude) at the visual point of camera, 
[Modification 2] 

In the above embodiments, the MR system of video 
see- through method is used to measure the attitude (or 
positional attitude) at the visual point, but the MR system 
25 of optical see- through method may be employed to measure 
the attitude (or positional attitude) with the image 
processing device of this invention. In this case, an 
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attitude sensor is attached on the HMD , and a camera is mounted 
on the HMD so that the relative attitude (or positional 
attitude) at the visual point position of the observer that 
is a measurement object is known. And the attitude (or 
5 positional attitude) of camera is calculated by the same 
method as in the above embodiments, and transformed to 
calculate the attitude (or positional attitude) at the visual 
point of the observer. The applicable scope of this 
invention is not limited to the object to be measured, but 
10 the attitude (or positional attitude) can be measured by 
mounting the camera and the attitude sensor on any object 
to be measured in a similar manner. 
[Modification 3] 

In the above embodiments, the template image is 
15 generated on the basis of the image 1° photographed at the 
predetermined positional attitude in a template image 
generating module 430. However, the template image may not 
be generated on the basis of the image 1°, but may be stored 
in advance or obtained by any of well known methods of updating 
20 the template dynamically, for example. 
[Modification 4] 

In the above embodiments, the neighboring area around 
the prediction position of landmark is only extracted as 
a target image in a target image generating module 404. 
25 However, the object image (i.e., target image) of template 
matching may not be obtained by extracting the neighboring 
area around the prediction position of landmark. For 
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example, an image I' consisting of an entire input image 
being rotated as shown in FIG. 14D may be set as the target 
image common to each landmark, the prediction position of 
each landmark being obtained within the I ' to make a 
5 corresponding search in its neighborhood, or over the entire 
area of the image I ' . 
[Modification 5] 

In the above embodiments, in order to measure the 
attitude or positional attitude, the detection of landmark 
10 by template matching as one means is employed. However, 
the landmark detecting method for the image processing device 
of this invention can be applied without regard to the 
measurement of the attitude or positional attitude, if the 
landmark position is detected from the image by template 
15 matching. 

[Other embodiment] 

It is needless to say that the object of this invention 
can be accomplished in such a manner that a storage medium 
(or a recording medium) recording a program code of software 

20 for implementing the functions of the above embodiments is 
supplied to the system or device, and the computer (or CPU 
or MPU) of the system or device reads and executes the program 
code stored in the storage medium. In this case , the program 
code itself read from the storage medium implements the 

25 functions of the above embodiments, and the storage medium 
storing the program code constitutes this invention . Hence , 
it will be appreciated that the functions of the above 
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embodiments may be not only implemented by executing the 
program code read by the computer, but also the operating 
system (OS) operating on the computer may perform a part 
or all of the actual processing on the basis of an instruction 
5 of the program code, thereby implementing the functions of 
the above embodiments. 

Further, it will be appreciated that the program code 
read from the storage medium may be written into a memory 
for a function extension card inserted into the computer 

10 or a function extension unit connected to the computer, and 
the CPU provided in the function extension card or function 
extension unit may perform a part or all of the actual 
processing on the basis of an instruction of the program 
code to implement the functions of the above embodiments. 

15 In the case where this invention is applied to the 

storage medium, which stores the program code corresponding 
to the flowcharts as shown in any one of FIGS. 5 to 13 and 
described previously. 

With the above description, it is possible to correct 

20 for the measurement errors at the visual point of camera 
with the attitude sensor, in particular, the accumulated 
error in the azimuth direction caused along with the elapse 
of time, whereby the MR without dislocation can be realized. 



